fix(gpu): tear down cuda-lts VHD prebake on GRID nodes [backport #8919 to v20260709] - #8924
Merged
Ganeshkumar Ashokavardhanan (ganeshkumarashok) merged 2 commits intoJul 13, 2026
Conversation
Ganeshkumar Ashokavardhanan (ganeshkumarashok)
requested review from
Chou Hu (AbelHu),
Devinwong,
Sri Harsha (SriHarsha001),
Nishchay (awesomenix),
Calvin S. (calvin197),
Cameron Meissner (cameronmeissner),
Sylvain Boily (djsly),
Karen Chen (karenychen),
lilypan26,
Mark Ibrahim (mxj220),
Peter Damianov (pdamianov-dev),
Patrick W. Healy (phealy),
r2k1,
Runzhen (runzhen),
sulixu,
Suraj Deshmukh (surajssd),
Tim Wright (timmy-wright),
Thibault Cohen (titilambert),
Xu Xue (xuexu6666) and
Zachary (zachary-bailey)
as code owners
July 13, 2026 22:42
Ganeshkumar Ashokavardhanan (ganeshkumarashok)
temporarily deployed
to
test
July 13, 2026 22:42 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
Copilot started reviewing on behalf of
Ganeshkumar Ashokavardhanan (ganeshkumarashok)
July 13, 2026 22:44
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Backports the GPU provisioning fix from #8919 onto official/v20260709 to prevent NVML driver/library mismatches on GRID (A10 / NVv5) nodes booting from CUDA-prebaked shared Ubuntu VHDs by tearing down a mismatched CUDA prebake before the GRID managed driver install.
Changes:
- Add
cleanUpGridNodeCudaPrebaketo remove CUDA-prebaked driver artifacts when the node’s requested managed driver kind is GRID and the VHD marker kind is non-GRID/legacy. - Invoke the new cleanup early in
ensureGPUDrivers(before the config/validate dispatch) so it applies to both managed driver install paths. - Add ShellSpec coverage for the new decision logic (teardown vs no-op, including legacy marker and non-Ubuntu no-op).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| parts/linux/cloud-init/artifacts/cse_config.sh | Introduces cleanUpGridNodeCudaPrebake and runs it early in ensureGPUDrivers on Ubuntu to avoid GRID vs CUDA prebake collisions. |
| spec/parts/linux/cloud-init/artifacts/cse_config_spec.sh | Adds ShellSpec tests validating teardown/no-op behavior for GRID vs CUDA marker-kind combinations and non-Ubuntu behavior. |
Copilot started reviewing on behalf of
Ganeshkumar Ashokavardhanan (ganeshkumarashok)
July 13, 2026 22:49
View session
Contributor
Changes cached containers or packages on windows VHDsPlease get a Windows SIG member to approve. The following dif file shows any additions or deletions from what will be cached on windows VHDs organised by VHD type.
diff --git a/vhd_files/2025-gen2-tl.txt b/vhd_files/2025-gen2-tl.txt
index a901175..5fc7b9e 100644
--- a/vhd_files/2025-gen2-tl.txt
+++ b/vhd_files/2025-gen2-tl.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
diff --git a/vhd_files/2025-gen2.txt b/vhd_files/2025-gen2.txt
index 4d7aa19..5c054c3 100644
--- a/vhd_files/2025-gen2.txt
+++ b/vhd_files/2025-gen2.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
diff --git a/vhd_files/2025.txt b/vhd_files/2025.txt
index 1441011..1fe4b78 100644
--- a/vhd_files/2025.txt
+++ b/vhd_files/2025.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1 |
Comment on lines
+1345
to
+1352
| # Keep only when the prebake is explicitly grid (matches this grid node). An empty marker kind is | ||
| # a legacy cuda prebake; a "cuda" marker is a cuda prebake -- both mismatch a grid node, tear down. | ||
| if [ "${m_kind}" = "grid" ]; then | ||
| return 0 | ||
| fi | ||
| echo "AKS_GPU_PREBAKE event=grid_cuda_prebake_teardown driver_type=${NVIDIA_GPU_DRIVER_TYPE:-} marker_kind=${m_kind:-none} node_kind=${node_kind} action=teardown" | ||
| cleanUpPrebakedGPUDriver | ||
| } |
sulixu
approved these changes
Jul 13, 2026
Ganeshkumar Ashokavardhanan (ganeshkumarashok)
merged commit Jul 13, 2026
da3471a
into
official/v20260709
18 of 30 checks passed
Ganeshkumar Ashokavardhanan (ganeshkumarashok)
deleted the
ganesh/grid-prebake-teardown-0709-backport
branch
July 13, 2026 23:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backport of #8919 to the
official/v20260709release branch (prebake-enabled VHD).Fixes
nvidia-smifailing with "Failed to initialize NVML: Driver/library version mismatch" on agentpool GRID / converged A10 (NVv5) nodes and NAP GRID nodes booting the CUDA-prebaked shared Ubuntu VHD.Root cause: the shared Ubuntu VHD prebakes only the
cuda(-lts)driver + a DKMS marker. The existingcleanUpPrebakedGPUDriveronly runs on nodes that do not install a managed driver. A GRID node installs thegriddriver on top of the stale prebakedcudamodule +/usr/bin/lib64libs → NVML mismatch.Fix:
cleanUpGridNodeCudaPrebake, invoked fromensureGPUDriversbefore theconfigGPUDrivers/validateGPUDriversdispatch, tears down the prebake when the node installsgridand the marker is not grid (cuda / legacy kind-less). Pure driver-kind mismatch; CUDA nodes untouched.Cherry-pick of the squashed main commit
947d2def64— verified the introduced patch body is identical to what merged onmain.Related
main: fix(gpu): tear down cuda-lts VHD prebake on GRID nodes before managed install #8919 (merged)official/v20260702backport: fix(gpu): tear down cuda-lts VHD prebake on GRID nodes [backport #8919 to v20260702] #8921Test plan
cleanUpGridNodeCudaPrebake(incl. non-Ubuntu no-op)🤖 Generated with Claude Code